Evaluation Methodologies in Information Retrieval

This paper reports on the Evaluation Methodologies in Information Retrieval Seminar [1] held from 27 October to 1 November 2013 at the Schloss Dagstuhl Leibniz Center for Informatics that is a world-wide renowned venue for informatics where scientists come together to exchange their knowledge and to discuss their research findings. Dagstuhl offers modern facilities and is located in the beautiful countryside of Saarland, Germany. Schloss Dagstuhl has previously hosted other seminars on information retrieval topics, e.g. the Seminar 09101 on Interactive Information Retrieval which took place in 2009, but this one was the first one devoted to the specific topic of information retrieval evaluation. The seminar was attended by 42 participants from thirteen different countries, including a large number of established researchers as well as some some promising young researchers, and also practitioners from industry. As it is well-known evaluation of information retrieval (IR) systems has a long tradition (see, for example, [2]), however, the test-collection based evaluation paradigm is of limited value for assessing today’s IR applications, since it fails to address major aspects of the IR process. Thus there is a need for new evaluation methodologies, which are able to deal with the following issues: